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BOTTLENECK  PROBLEMS,  FUNCTIONAL  EQUATIONS,  AND  DYNAMIC  PROGRAMMING 

Richard  Bellman 


§1 .  Introduction . 

The  study  ol*  inter-industry  processes  Is  one  to  which  a 
great  deal  of  attention  has  been  given  in  recent  yeai's  .  l.i  pai*— 
ticalar,  it  has  become  a  matter  of  more  tncun  theoretical  interest 
to  determine  the  most  efficient  utilization  of  liiulti— industrj 
complexes . 

In  this  paper  of  essentially  expository  nature  we  sniall 
present  a  new  tneoretlcai  techniqw.e,  based  ipon  ine  principles 
of  the  theory  of  dynamic  prot^ra.uning,  which  may  be  used  to  treat 
some  of  the  novel  types  of  mathematical  pr^/blems  wiiicn  arise 
from  these  studies.  Subsequently,  a  paper  will  be  presented 
containiXig  the  solutions  to  a  number  of  specific  problems,  cased 
upon  Joint  work  by  the  autnor  ana  Sherman  Lehman. 

_Ihe  problem  we  shall  discuss  is  a  "bottleneCiC  problem"  in 
the  sense  that  the  level  of  economic  activity  will  be  aetemlneu 
by  tne  resource  in  shortest  supply^. 

To  begin  with  we  shall,  in  the  next  sectlori,  present  a 
typical  problem  of  the  bottlenecK,  type  involvinc:  the  auto  indus¬ 
try,  the  steel  Industry  and  tne  tool  inaustry.  Following  this, 

; 

we  shall  formulate  the  problem  .'natnematically  employing  a  dis¬ 
crete  approximation.  After  a  snort  alscussion  of  tue  ulffic  dties 
of  this  approach  wo  snail  turn  to  a  continuous  approximation.  We 
will  now  be  in  a  position  to  apply  tne  funcLlonal  equation  approuen 
ef  tne  theory'  of  dy^namlc  programmin,.: .  This  met.uou  .^seu  v/ltn  falUi 
and  resolution  will  often  yield  the  solution  quite  rapluiy. 
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To  prove  in  a  simple  fashion  tnat  what  we  nave  la  actaally 
a  sol  it ion,  we  exploit  the  linearity  of  the  problem  and  con¬ 
struct  a  dual  problem.  This  aual  problem  may  be  '.4sed  both 
to  verify  and  derive  a  solution. 

To  Illustrate  these  tecnniques  we  snail  consider  in  detail 
the  problem  of  deteminin-;  tne  yi'. and  ya(t)  wnich  maximize 
Xa(T)  given  tne  equations 


(1) 

-  aiyi(t)  xi(0)  -  Cl  , 

P- 

»  (a2-l)y2(t )-yi{t),  xa(0)  -  ca  , 

and  the 

constraints 

(2) 

(a) 

J\,  ya  >  ^  • 

(b) 

yi  +  ya  <  X 

(c) 

ya  <  ba  xi/aa 

(d) 

Xa  >  0 . 

Tills  problem  is  a  particularly  simple  example  of  the  mate— 
ematlcal  problems  wiilch  inter— Inaustry  analyses  raise. 

Finally,  we  shall  briefly  Indicate  tne  extension  of  our 
metno'is  to  otner  t,  pes  of  vai’iat ional  proulems  wiUci  Involve  non¬ 
linear  functions  ana  functionals.  For  those  wno  are  Interested 
in  other  aspect. s  cf  tne  tneor;;  of  d;.  naunlc  proyra.mming,  we  I'efer 
to  tne  papers  [l]  -  [3j  listed  in  t  ic  blDlioyraplv.  . 


§2 .  A  Typical  Problem. 

Let  us  consider  a  model  of  a  three— industry  production 
process,  where  the  individual  Industries  are  lue  auto  industy, 
the  steel  Industry, and  the  tool  Industry,  These  industries 
are  to  be  used  to  produce  as  many  autos  as  possible  over  a 
given  time  period  of  length  T. 

To  simplify  matters  we  shall  assume  that  eacu  inaustiy  is 
characterized  at  any  tine  t,  which  to  begin  witn  we  snail  allow 
only  to  take  tne  values  0, 1 ,2, . .  .  ,'i',  by  its  capacity  and  its 
stockpile.  Let 

(1)  Xi(t)  •  number  of  a-*tos  produced  up  to  time  t 
xa(t)  »  capacit.  of  auto  factories 

X3(t)  =  stockpile  of  steel 
X4(t)  *  capacity  of  steel  mills 
X5(t)  ■  stockpile  of  tools 
Xa(t)  -  capacity  of  tool  factories 

We  shall  assume  a  linear  production  pi-ocess  in  tec  sense 
that  output  is  always  dii'ectly  proportional  to  the  mininvum 
input.  Thus,  production  is  directly  proportional  to  capacity 
whenever  there  is  an  abundance  of  raw  materials  and  directly 
proportional  to  the  quaiitity  of  raw  materials  allocatec'  when¬ 
ever  there  la  an  abundance  of  capacity.  To  be  more  precise,  we 
postulate  that 

(2)  a.  An  increase  in  auto,  steel  or  tool  capacity  xequli-es 

steel  and  tools 

b.  Production  of  autos  req.ires  auto  capac.  l.  ^nd  otee] 
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c.  Production  of  steel  requires  only  steel  capacity 

d.  Production  of  tools  requires  tool  capacity  and  steel. 

Tne  process  proceeds  in  the  following  fashion.  At  tne 
beginning  of  eacn  unit  time  period,  say  t  to  t+^,  we  allocate 
various  quantities  of  steel  and  tools,  taicen  from  t  ieir  res¬ 
pective  stockpiles,  for  the  purposes  of  producing  autos,  steel, 
and  tools  and  of  increasing  the  auto,  steel,  and  tool  stockpiles. 

Let 

{j>)  (a)  z^(t)  ■  amount  of  steel  allocated  at  iin.e  t  for 

the  purpose  of  increasing  x^(t), 

(b)  w.(t)  -  amount  of  tools  allocated  at  time  t  for 

the  purpose  of  increas.^ng  x,(l). 

Upon  referring  to  tne  ass'i/'.pl Ions  in  (2),  we  see  tnat 

(4)  (a)  -3  «  0 

(b)  wj  »  W3  ■  v/s  »  0  . 

Combining  tne  ass'umptioris  in  (2)  wit;,  tnose  of  tne  pr‘'vio  <s 
paragraph,  we  obtain  the  following  equations  whic;';  relate  Xj^(t  +  l) 
to  x,(t),  z^(t)  and  w^(t): 

(>)  Xi(t  +  l)»>ti(t)+Mlri  (^’iXa(t),  c*iCx(t),  ^iWi(t)) 

Xavt  +  l)  B  xa(t)  +  Mir;  ('^a'^a^t),  ^aWa(t)) 

X3(t  +  l)  *  X3(t)  —  iii(t)  -  ;-a(t)  —  ^4(t)  —  <^5(1)  —  ^a(t)  +>3X4(1) 

X4(t  +  1)  »  X4(t)  +  M5n  (c»^S4(t),  ^♦W4lt)) 

X5(t  +  1)  »  X5(t)  -  W2(t)  -  W4(t)  -  Wo(t)  +  M.n  [>'5X9(1)  ,<A3-s(t)] 

Xo(t  +  l)  =  Xo(t)  +  Min  (akoZo(t ) ,  PoWa(  t )  )  , 

wnere  and  are  conui.ants. 


The  constraints  upon  and  are  obviously 


(O) 


(a)  >  0 

(b)  Zj  +  Z2  +  Z4  +  Z5  +  Zo  ^  X3 

(c)  W2  +  W4  +  Wa  <  X5  , 


together  with  tne  "comiaon  sense"  constraints 


(7) 


(a) 

•^iZi 

m 

^iWi  <  ^1X2 

(b) 

=*aZa 

- 

?aW2 

(c) 

^424 

m 

94W4 

(d) 

< 

^sXa 

(e) 

«^aze 

s 

^aWa 

Bj'  means  of  these  aaoitional  constraints  we  may  elininate  w. 

i 

completely,  obtaining  in  place  of  (j)  tne  system  of  equations: 

(8)  xi(t+l)  »  xi(t)  +  ^iz,{t),  xi(0)  »  Cl  , 

X2(t4l)  =  X2(t)  +  ^2Z2(t),  X2(0)  =  Cg  , 

X3(t  +  1)  -  X3(t}-Zi(t)-Z2(t)-Z4(t)-Z5(t)-Ze(L)-f  ^2X4(1)  ,  X3(0)=C3  , 
X4(t  +  l)  =  X4(t)  +  =^4Z4(t),  X4(0)  =  C4  , 

X5(t  +  1)  =  Xs(t)-e2Z2(t  )-€4Z^(t)_foZQ{t)+0^25(t),  €  .  »  ol  / &,  , 

i.  X  .i. 

X  5  (  0  )  »  C  5  , 

Xe(t  +  1)  =  Xo(t)  +  '=<aZo(t),  Xo(0)  =  Ce. 

Tne  constraints,  in  turn,  nave  tr.e  I’oria,  for  eacn  t: 

(9)  (a)  >  0 

(b)  zi  +  Z2  +  Z4  4  Z5  4  zo  <  X3 

(c)  <222  +  ‘424  4  <e2a  <  X5 

(d)  Zi  <  f2X2 
(^) 


25  f  eX  a  • 
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Wn  must  now  choooe  the  z^(t)  for  t  ■  0, 1 ,2, . . . ,T— 1 ,  subject 
to  the  above  constraints,  so  as  to  maximize  Xi(T). 

§3 •  Discussion . 

As  fomi^lated,  the  problem  above  lies  witnln  the  domain  of 
linear  programii.ing .  The  problem  of  maximizing  Xi(T)  may  be 
solved  for  any  given  set  of  constants  by  a  straightforward  iterative 
process  of  the  type  developed  by  G.  Dantzlg  and  others. 

A  rapid  count  of  the  number  of  variables  Involved  for  T  of 
moderate  size,  say  ^0,  will  show  that  even  this  simple  three- 
industry  problem  with  lumped  capacities  and  stockpiles  leads  to 
a  problem  of  huge  dimensions.  If  we  wish  to  determine  the  form 
of  trie  solution  corresponding  to  different  initial  states  and 
corresponding  to  different  sets  of  constants  of  proportionality, 
we  are  faced  with  a  numerical  task  of  vast  magnitude.  This  type 
of  analysis  of  the  solution,  a  "sensitivity  analysis,"  is  required 
whenever  we  make  the  crude  assumptions  of  lumping,  linearity  and 
so  forth  made  above  • 

The  question  then  arises  aii  to  whe'v^her  or  not  it  Is  possible 
to  determine  the  Intrinsic  structure  of  an  optimal  policy,  " 

less  of  any  numerical  values  which  we  may  subsequently  assign. 

Ihis  Knowledge  is  not  only  of  impcrtante  in  itself,  but  is  also 
of  the  utmost  impurtance  in  determining  approximate  solutions  in 
cases  wnere  any  explicit  analysis  is  too  difficult. 

For  further  discussion  of  this  point  we  refer  to  [ji]  ,  . 
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§4  ,  A  Continuous  Version. 

It  is  well-4cno«m  In  the  physical  world  that  continuous 
approximations  are  more  amenable  to  mathematical  analysis  than 
discrete  approximations ,  that  calculus  is  a  more  powerful  tool 
than  algebra.  Let  us  follow  this  lead  and  derive  a  continuous 
version  of  the  equations  in  (2.8)  and  (2.9). 

In  passing «  let  us  note  that  it  may  very  well  be  that  the 
continuous  version  we  discuss  may  correspond  more  closely  to 
the  actual  economic  problem  than  the  discrete  versions  which 
have  heretofore  been  treated. 

In  the  continuous  version  of  the  problem  the  allocations 
Zl(t),  over  the  time  Interval  [t,t+l]  are  replaced  by  the 
allocations  z^(t)  t  over  the  time  Interval  [t,t+At] .  The 
quantities  z^(t)  are  now  rates  of  allocation.  Writing  out 
the  equations  corresponding  to  (8)  and  letting At>0,  the  new 
equations  take  the  form 

(1)  Ai(t)  -  '^izi(t) 

xs(t)  -  '*sza(t) 

X3(t)  -  -Zt(t)-Z3(t)-Z4(t)-Z5(t)-Ze(t) 

*4(t)  -  ^4Z4(t) 

*8(t)  -  -««Za(t)-C4Z4(t)-€eZa(t)+e^Z5(t) 

*s(t)  -  «^aZe(t) 

The  constraints  upon  the  z^^  are  now 

(a)  z^  >  0 

(b)  Zi+Zt-*-Z4+25  +  Za<  00 

(c)  €,Zf+€4Z4+€aZe<  oo 

(d)  z,  ^  faXt 

(e)  Zg  <  leXc 


(2) 
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This  means  that  the  constraints  of  (2b)  and  ^c)  disappear 
and  are  replaced  b; 

(^)  (b' )  X*  2 

(C  )  xi  >  0, 

inequalities  idiich  were  obviously  satisfied  previously  • 

Ihe  importance  of  these  new  constraints  resides  In  the  fact 
that  when  xa  ■  0,  we  must  have 

(4)  Zi  -t-  za  Z4  zs  zc  ^  r»  X4 
and  similarly  when  Xs-O  we  must  have 

(5)  +  ^424  +  *o2o  <  <i4iZa  . 

Subject  to  the  above  equations  constraints  we  wish  tv 
maximize  Xi (T) • 

§3 .  Some  Rema^-ks  on  Notation, 

Before  presenting  our  general  theoretical  approach,  let  us 
introduce  a  small  ajr.ount  of  vector-matrix  notation.  Let  x(t) 
and  z(t)  denote  n-dlmenslonal  column  vectors 


and  A^,  for  such  values  of  1  and  J  as  occur,  denote  n  x  m 
matrices . 


P-.P,;, 

— 


ly  D  non-ne^at '..VC  vector  x(t)  we  shall  mean  one  .’or  whlviP. 
all  or  the  ■'o:v''on''nt3  are  non— cc^^at V/e ,  i.e.,  x,  >  0,  and  write 

X  — 

X  2  C*  The  Inequalltv  x  >  y  la  to  be  equivalent  to  x--”  non- 
negat  Ive  .. 

In  terns  oT  thin  notation  the  above  equal  . 'ions  and  conn ir''.  into 
have  the  form 

(2)  (a)  AgZ,  a(0)  ■  G, 

(b)  z  >  0 

(c)  BiZ  ^  Btz. 


To  express  our  criterion  function  In  simple  fom,  let  us 


Introduce  the  vector  inner  product 


n 


(3) 


(a)  (^#y)  -  ZZ 


x^y. 


We  .nay  tnen  write 


{^)  x»(T)  -  (x-(T),ot)  , 


Wiiere  the  first  component  of  cA  is  l  and  the  remaining  components 
are  0. 

Finally,  we  shall  write 

(5)  Sup  (xCT),*)  -  r(c,T)  , 

-T 

where  f  is  a  scalar  f.n'^t'on  of  t.io  vecter  r-  *  .ne  t.’.me  variable 

T,  i  .e  . , 

(6)  f(c,T)  =  r  ( c j , Ja » • • • » ; T)  . 

Observe  that  we  use  f-ap  In  (h)  since  the  maxi-mum  may  not  be 
attained.  We  sno.  I  'iiooui’c  tn^o  Point  again  below. 
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§6 .  ba3l:  j^anctlorial  Eq^atlori. 

in  wi’Liirit;  (D-t>)  we  have  tacitly  made  use  of  tne  extreriely 
Irijjoriant  '.iot  tnat  t.\o  optimal  yield,  Xi(T),  !□  a  function 
only  of  tnj  initial  state  and  the  duration  of  the  process,  once 
tne  lawa  (jovernlnc  the  process  have  oeen  codified, 

l*et  u3  farmer  observe  tnat  tne  nature  of  an  optir.ial  al¬ 
location  policy  Is  such  that  its  continuation  over  a  sub— Interval , 
say  [S,!]],  must  be  an  optimal  policy  for  a  process  of  duration 
T-S  startinc  from  the  Initial  state  c(S). 

The  mathematical  trani llteratlon  of  this  truism  will  yield 
the  basic  functional  equation,  namely 

(1)  f(c,Svr)  -  Max  f(c{S),T)  , 

ro,r.] 

where  by  Max^  we  mea,!  maximization  over  all  z(t)  for  0  <  t  <  6, 

i*al  1/ f,,  ife  .■<)n..t  ra Ini  ? .  Let.  .5  f-  r  oir'.pllclt:  ase-ome  irv 
iriomeni  M  ‘'o  uhtxltn'tr.  is  attained. 

Inf  initesiir.al  Analo^^ne  . 

Let  U3  now  assume  that  c(S)  ^  x(S)  possesses  piecewise 
continuous  derlval^'cc  urKt  >.btt  f  has  the  same  property.  Then 
mining  S  =  0  to  €  well-behaved  point,  've  have, 

(1)  X(S)  -  t  4  [AjC  +  A2Z(0)]8  4-  0(S)  , 

C4. 1  i 

(.  )  Uc.',  £.T)  ..  ;  .o,T)  +  3  +  0(S) 

Fic(S),T)  -  .  (  ■  [A,.:  +  Aa  :(0)]i.,T) 

-  .'c,;')  ‘  3;a.c  -  a,2(0),  ^  )  +  0(S), 


1  ^ 


where 

(^) 

Sutstl'.'jl-ln In  (6.1)  and  let!,  S  -^0,  we  see  that  In— 
flnlteslina]  ^:eneraLoi-  cf  (l)  is 

(‘0  3T  i  Aaz(O),  , 

z(0)  L  J 

where  the  inajcir,.un  Is  now  laker  nvev  an  n— d  In*  ns  Icnal  reglo;  ^ 

delennlned  l./  she  coiui  r‘a ants  . 

'i’yie  L-iporiance  ot'  the  inrii\itc'3:^i.ai  anaiojuc  Is  ina-  11 
pernlca  uj  to  ctetenaino  ii.e  coiutlon  over  [0,T-»-aTj  if  we  have 
it  detemin^d  over  [o,t3  for  all  Initial  states.  Since  Iri 
most  of  these  prooivai.'  ’-'.e  i  1  J'f icultiea  are  reauliy  resoivea 
for  araaii  V,  we  nave  a  feasibi'*  approacii. 

As  an  11  lustral  J  cr.  of  tj.rrse  ron.  .’K-h  we  vf  iall  plou^.n  .h.r\  a£;fi 
the  detail.:  f  a  ivlatr-v;^  simple  problem  below. 

§8 .  A  Particular  rroLlttr.. 

Let  U6  MOW  concldoi*  t.he  {Toble;  of  na>:imlzlng  Xa(T)  given 
the  following  equavicns  and  constraints 

^  ^  ^  cT^  ~  ^iv6)  '  I  » 

■  aa^»— tj,  xa(0)  *=  cg  f 


where  Zi  and  zg  are  subject  to  the  constraints 


(2)  (:i)  -1,7.2  >  0 

(b)  zt  zs  <  xa 

(c)  za  <  Xi 

(d)  xa  >  0. 

(^)  xa('^’;  =  i’(Ci,Ca,T). 

[o/'l 

it  Is  easy  ’0  s^e,  using  either  classical  theorems  In  the  calc^-ljs  of 
/arlaiiono,  or  a  weaK  convergoacc  argument,  that  the  maximum  lo 
aaaumed  in  Lrils  case. 

As  In  r  satisfies  tiie  iunctlonal  equation 

(■•;  i’Ui.ci,,  SH)  -jVUx  .•'{x,{S),x«(S),T)  , 

[O.S] 

wiilcii  ie.iun  -o  uj  In  §7#  tn«  noil— li.near  partial  dllfei’antial 
c'l  latiin 

*  *  *  ^' *^*~^‘*  ^*  J  ■ 

i'.i..  ;..ax^muv.  J.o  u-i’  the  region  affined  by 

(u)  (a)  0  <  zi,  za 

(-')  -a  <  '••a 

(c)  Za  <  Cl  , 

with  the  additional  restraint 

(7)  aaZa  —  Zi  >  0 

If  X2  -  0 . 


l.et  u3  now  sketch  the  analytic  pfocea^re  that  v;lll  ylelJ 
a  solution.  Wc  bc^in  with  the  moat  conplioateu  case,  that 
•.irtiere  c#  <  Ci.  .  Por  a  process  o:’  short  deration,  the  solution 
la  trivial.  We  have 


(S) 


a  I  *0,  Zg  *  Xg  , 

agT 


f  *»  Cge' 

Tr*lB  policy  is  purauau  uiitil  a  "Lottlenecic"  develops, 
wnlch  is  to  say  cg  exceeds  c^.  the  optimal  policy  uesci'ibeu 

in  (8)  we  see  that  this  situation  will  occur  as  soon  as  T  exceeds 
T|  ■  log  (ct/cg)/sig  . 

To  obtain  the  solution  Tor  1  >  Ti,  we  rewrite  in  the 

form 

(9)  *f  _ 


” '  ”ui)  t'*®’  ^  ~  ^  ‘'*3 


The  location  of  tne  maximizing  point  (zt(0),  zg(0))  will  depend 
open  the  sign  and  rnagnitud*-  of  the  coefficients  of  and  ga« 
Por  T  <  Ti  we  have 


MO)  a.  ^  ^  -  e®**^ 

<9c»  ® 


Bg 


>T  , 

at 


Using  o’.r  asB'np'.icn  concerning  the  continuity  of 


_  £f_ 

?ci  »  aca 

we  suspect  tnat  the  solution  for  'i  sligntiy  gi'eater  than  will 
have  the  foxsu 

(a)  zi  •  0,  za  *  xa  for  0  <  S  <  Ti 
Zi  =0,  L'a  Xi  for  Ti  <  S  ^  T 


(11) 


Applying  this  pollcv  ,  r  tEKea  tne  fom 


(12)  r  «  baOl  (T— T|)  agbaCt  * 


where  Tt  la  aa  above.  In  order  to  determine  how  lon^^  thie 
policy  endureu  when  T  >  Tt,  we  consider  the  process  as  starting 
I'rom  S  ■  T|.  In  terns  or  ct  »  ct('i|}»  ca  ■  tta(Tt)f  T  has  the 
form 


(15)  f  •  Ca  aacl(t— Ti). 

Tlie  eq-xatlon  vrtiSch  replaces  19)  l.a?  precisely  the  same 
I'om;  with  ct,C2  I'ep laced  by  ca»  namely 


=  (TJ 

mfe  have,  using  (15), 


(lr>) 


**  ITT 


if 

9ca 


aiSavT— Tt )  —  1 , 


ai 


Jf 

J?T 


u.a 


Tr.c  coefi'lc . err  of  Zj  Ja  negative  for*  T  <  T*  *  Tt  +  i/aiaa, 

0  at  T*,  and  positive  thereafter. 

It  follows  that  the  new  policy  given  by  (11)  remains  optimal 
for  Tt  <  T  <  T*. 

FurtneiTjoro  since  'i'*— Tt  is  Independent  of  ct  and  ca  ee  eee 
tnat  we  Know  tne  fora  of  the  optimal  policy  over  a  tail  interval. 

It  remains  to  determine  what  the  policy  is  in  the  middle  in 
tne  general  case  when  T  exceeds  T^.  s'e  suspect  that  it  has  the 
form 


(16)  2i  ^  Xa  -  X|  ,  za  -  xi 


••  • » 


III  r)laci  of  '/erll’ylfij  tills  a'urecUly,  vrtiicn  may  bo  Jorie, 

'70  shi'll  ioaorlbo  In  the  next  ooctlon  a  more  eiegant  toc.vnique 
'.7hlcn  explo.lto  the  linearity  of  tne  equatlona. 


§9*  A  Dual  Problem, 

Tjft  as  take  oar  be.ilc  eqaatlons  to  have  the  fonn 

(1) 


^  •  Ai,  z{0)  ■  o, 
wi&h  constraints  of  tnc  fom 


(2) 


(a)  z  ^  0 

(b)  Bz  ^  X 


u 


since  X  •  c  y'  Azdt,  tae  constraint  nay  be  wrltte 
o 

t 

())  Bz  +  y*  Czdt  ^  c,  (C  ■  -A)  . 


The  pr’obieM  of  '..uclmizi-n;  [f.  Is  equivalent  to  tnav  of 

T  T 

maxL'iLzln^  J  tAz,<A)dt  -  j  (z,di.')'i'-  • 

o  o 

Beginning  aii  over  again,  we  start  with  the  problem  of  max- 
T 

Imlzing  J  a  y’(z,3t*)ot  over  ail  z  satisfying  tne  cohstrainta 
o 


(^) 


(a)  z  2  0 

(b)  hz  ■¥  J  C*cit|)clt  <  c 


Let  w(t)  be  a  tion-nei^atlve  vector  of  the  sane  ninenoion  as 
c.  Then  by  virtue  of  {^b),  we  na/e 

(S)  J  ^  Czdti)<it  ^  ^  (w,c)<it. 
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Let  B'  denote  the  transpose  of  B.  Then,  as  Is  easily  seen, 
(Bz,w)  ■  (z,B'w).  Integration  by  parts  yields,  for  any  constant 
matrix  C, 


(0) 


/*  (w,  yC2dti)dt  ■  r{  S  C*  wdt,2)dt. 
0  o  o  t 


Combining  these  two  results,  we  nave 

T  t  T  T 

(7)  J'  {w,Bz  +  C  Czdt|)dt  -  J'  (B*w  +  f  C'wijt,z)dt 

O  0  o  t 

Let  us  now  assume  that  it  is  possible  to  flnu  a  vector 

w  »  w(t)  which  Is  non— negative  and  satisfies  the  inequality 

T 

(8)  B'w  +  ^  C'witi  > 

We  tnen  have  the  cnain  of  equalities  and  inequalities 

T  T  t 

(9)  C  (w,c)dt  >  y  (w,B2  +  y  Czdti)at  » 

0  “  o  0 


T  T  T 

■  y  (B'w  +  y  c*wdt,z)dt  >  y  (o<',z)dt  . 

O  t  0 

From  this  it  is  clear  that 

T  T 

(10)  Inf  y  (w,c)dt  >  Sup  y  (z,ok')dt  » 

wo  z  0 

where  tne  Infimum  and  supremum  are  taxen  over  all  w  and  z 
satisfying  the  inequalities  of  (8)  and  (4b).  If  the  minim.um 
and  maxinrum  are  assumed,  tiie  details  are  as  above.  If,  however, 
the  minimu:n  and  maximum  are  not  assui’ed,  then  delta— functions 
will  occur,  which  is  to  say,  we  must  refonriulate  tne  problems 
in  tenns  of  Stleltjes  integrals.  A  number  of  interesting  and 
difficult  problems  arise  in  this  way,  which  we  shall  not  discuss 


here . 
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If  the  two  extremes  In  (10)  are  equal,  we  see  that  tt^e 
f'''* lowing  relations  must  hold,* 


(11) 


t 

0  If  >  (Bz  +  r  Czdt)^ 

•  °  T 

0  If  <  (B'W  +  f  C'wdt)  . 
*>  t 


The  Important  fact  which  we  now  wish  to  establlsn  is  that, 
conversely,  any  pair  of  non— ne^atlvci  ^  and  w  satisfjrinc;  (11) 
and  the  original  constraints  will  furnish  solutions  to  the 
maximum  and  minimum  problems . 

To  demonstrate  this,  let  us  note  that  if  (12)  holds,  all 
the  relations  In  (9)  are  equalities.  Assume  now  that  z  is  anoi;  cr 
vector  satisfy irife  all  th--'  constraints  anu  for  wnich 

T  T 

(12)  /  (z,<*.')dt  <  r  • 

Tiien  with  the  w  associated  with  z  we  have 


(l:^) 


(z,ot'  )dt  < 


o 


T 

(z,  B’w  +  f  A'wdti)dt  ■= 
t 


Azdti,w)dt  < 


(c,w)<it  = 


T 

o 

a  contradiction. 


« 


Apart  from  sets  uf  ineatuie 


zero,  to  be  technical  . 


-18- 


II  follows  then  that  we  have  a  systematic  orocedare  for 
verify Imj;  a  conjectured  solution.  Given  z,  we  detemlne  w  by 
means  of  (11),  and  then  see  whether  or  not  w  satisfies  the 
given  constraints.  In  the  next  section  we  shall  carry  through 
the  details  for  the  problem  of  §d. 

§10.  Verification  of  the  Solution  Given  in  §8. 

Applying  the  techniques  describea  above,  we  find  that  the 

dual  of  the  problem  proved  in  §3  is  tnat  of  minimizing 
T 

J'  (ciWi  +  ctWa)dt,  where 
o 

(1)  4  wa»  yi(T)  =  -1  , 

*  -agwa,  ya(T)  -  aa 

and  the  constraints  have  the  form. 


(2)  (a)  wi,w,  >  0 


(b) 

Wl 

+  Wa  >  y« 

(c) 

wa 

The 

eq  lations 

of 

(9.11)  are 

now ; 

If 

(^) 

(a) 

za 

<  Xi, 

then 

Wl 

B  0 

(b) 

Zl 

+  za  <  xa, 

then 

Wa 

•  0 

(c) 

Wa 

>  yi» 

then 

2  1 

-  0 

(d) 

Wl 

+  Wa  >  ya. 

then 

Za 

■  0  . 

p- 


We  have  oaltted  the  condltlono  coiTesponalng  to  x*  ^  0 
3lncc  we  3  ispGCt  that  the  optional  allocation  policy  aatomatlcally 
keepo  Xj;  >  0. 

We  wish  to  verify  that  the  pollCj  which  maximizes  x(T)  i6 

(4)  (a)  For  T-l/aia*  <t<T,  zi(t)*0,  za-  Min  (xx,xa) 

(b)  For  0  ^  t  <  T-l/ajaa,  (1)  if  xa  <  Xi ,  Zi  »  0,  za  =  Xa 

{2)  ii‘  Xa  >  Xi,  zi  --  Xa-Xi 

®  X  ^  • 

It  is  easily  seen  that  this  is  a  pemlssible  policy  in 
that  Zi  ■  Xa^i  i3  actually  non-negative  when  zi  and  za  liave  the 
above  values. 

Having  prescribed  z,  v;e  caii  icter.uine  w  using  {^)  and  then 
test  for  consistency.  There  are  two  cases  to  consider,  depending 
upon  whether  Xa  ever  exceeds  Xi  or  not. 

Let  us  assume  then  that  T  >  T^,  in  which  case  Xa  can  exceed 
Xi  If  appropriate  policies  are  used. 

Case  I :  T  -  l/ajaa  <  Tj<  T.  The  sol  .tlon  is  given  by 

(3)  for  t  <  :  Zi  =  G,  iSa  “  xa 

for  t  >  :  Z|  »  0,  Z£  «  Xi  . 

For  t  <  Ti  these  results  ylelu.  In  conjunctloi'.  witu  (3)» 

(6)  for  t  <  T^,  wi(t)  -  0,  Walt)  =  yalt.) 

for  t  >  T^,  Wa(t)  -  0,  Wi(t)  =  ya ( t )  . 

For  t  >  we  obtain,  using  (1) 

(7)  y2(t)»aa#yi(t)»-l+  aiaalT— t)  <  0, 
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while  for  wo  nave 

Id)  y.Cx)  -  ^  j  ^ 

y,it)  -  -1  +  a,aa(T-T,)  -  <  0 

Hence,  the  Inoqualitlea  Wi,Wa  >0,  Wa  >  ji,  Wi  >  ^a  are  satia— 
fied  in  their  respective  intervals. 


Ca.oe  II.  Ti  <  T  —  l/e-iSa.  Tnis  is  Liio  most  interesting  case. 
The  vectors  z  and  w  are  now  detemined  as  follov/s: 

(9)  for  T-l/aiaa_^t<T:  Zi-O  wa=0 

*  xtf  wi  »  ya 

for  Tq  <  t  <  T  -  l/a^aa:  Zj  »  Xa-Xi  Wa  -  yi 

Zi  «  X 1,  Wi  »  ya  -  y  I 

for  Zi«0  Wi-0 

za  ■  Xa  Wa  =  ya 


For  T  —  1/aiaa  <  t  <  T  we  have 

(10)  ya(t)  -  aa,  yi(t)  =  -1  -t-  aiSalT-t). 

Hence,  in  this  'nterval  yi(t)  <  0  •  wa .  Note  tnat  yi(T— l/aiSa)  =  0. 
In  the  range  Tq  <  ^  T  -  1/aiaa,  we  have  the  equations 

•  — aaya  ^ 


“^a/a 


Let  J.B  show  that  yx  >  0  ana  ya  >  yi  this  range.  Starting 
from  t  -  T  —  l/aiBa  where  tne  inequalities  are  satisfied,  let  ns 
reverse  the  tljae.  The  bac.<waru  equations  are 
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(12) 


^  -  a«ya  -  (l-*-ai)yi 

^  -  a«y« 


Px*om  this  we  obtain 


(1^) 


^  (yr-yi)  -  (l-^ai)yi 


Hence,  If  yi  remains  non-negative,  we  will  have  yt— yi  >  0. 
It  Is  clear  that  dyt/dt  starts  out  positive  and  sta^s  positive 
as  long  as  (yi,ya)  remains  above  aaya  —  (l'*’ai)yt  *0.  If  it 
hits  the  line  we  have  dyi/dt  -  0,  which  means  that  y^  has  a  maxi¬ 
mum  or  a  point  of  Inflection.  Both  are  excluded,  since 


(u) 


iiu  .  a,  _  (i+„)  iu  .  aj  y,  >  0. 

dt*  dt  dt 


Ihla  shows  that  wt  and  wa  remain  non-negative  in  this  interval 
Finally  for  t  <  we  have 

^  -  y**  ^  •  -atys  . 

As  t  decreases,  ya  Increases  and  yi  decreases.  Hence,  ya  >  yi 
remains  valid. 

This  completes  the  verification. 


§11.  Won-Llneau*  Problems. 

A  great  number  of  problems  In  matnematlcal  economics  reduce 
to  the  maximization  of  an  Integral  of  the  form 

(l)  T  "  J  ^(*4 **s»  •  •  •  ,za, . . .  ,z^)dt 
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and  there  a  number  of  constraints  of  the  form 

(3)  •  •  •  »Jtn*  »^a»  •  •  •  »Zj^)  ^  0  * 

cf.  [].  n  . 

These  problems  may  also  be  approached  by  the  functional 
equation  outlined  above.  A  brief  outline  of  the  procedure, 
together  with  an  extension  to  eigenvalue  problems,  will  be 
found  In  [  }  . 


# 


-2^ 


Bibllosrapn\ 


1.  R.  Bellman,  An  Introduction  totfae  Theory  of  Dynamic  Programming, 


roceeainge  oi  cne  National  acaaemy  ox  science 
Vol.  j>9»  No.  9»  September  1953* 

utatlonal  Problems  In  the  Tkxeory  of  Dynamic 


paper,  dymposi-ra  on 

umericai  analysis,  Au^^ust,  19b^«  Santa  Monica, 
California 

SooM  Problems  in  the  Theory  of  Dynamic  Programmi 


conometrica,  (to  appear;. 


i!UiSfiE5QU[ 


I’oceases ,  naNU  paper, 
oumai  oi  nathemalics,  (to  appear) 


R.  Bellman,  I.  Ollcksberg,  0.  Gross,  On  Some  Variational 

Problems  in  the  ‘Iheory  of  t^namlc  Proyammine, 
Proceedings  of  the  Rational  Ac ademi'  or  Science, 
Vol.  :>9,  (1953). 

R.  Bellman,  S.  Lehman,  On  the  Continuous  Qold  Mining  Equation, 
RAND  paper,  P— 43b,  Proceedings  of  the  National 
Academy  of  Science,  (to  appear). 


